Adaptive Behavioral HTTP Flood Protection

ABSTRACT

A system and method to detect and mitigate denial of service and distributed denial of service HTTP “page” flood attacks. Detection of attack/anomaly is made according to multiple traffic parameters including rate-based and rate-invariant parameters in both traffic directions. Prevention is done according to HTTP traffic parameters that are analyzed once a traffic anomaly is detected. This protection includes a differential adaptive mechanism that tunes the sensitivity of the anomaly detection engine. The decision engine is based on a combination between fuzzy logic inference systems and statistical thresholds. A “trap buffer” characterizes the attack to allow an accurate mitigation according to the source IP(s) and the HTTP request URL&#39;s that are used as part of the attack. Mitigation is controlled through a feedback mechanism that tunes the level of rate limit factors that are needed in order to mitigate the attack effectively while letting legitimate traffic to pass.

RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application60/828,771 filed Oct. 9, 2006, which is incorporated herein byreference.

BACKGROUND OF THE INVENTION

1. Field of Invention

The present invention relates generally to the field of computersecurity. More specifically, the present invention is related to asystem and method providing adaptive behavioral HTTP protection againstHTTP floods attacks that misuse the resources of Web servers.

2. Discussion of Prior Art

Applicant's pending application titled “Dynamic Network Protection”teaches a method for protecting a network from an attack wherein themethod involves measuring a property of traffic entering a network,analyzing the property of traffic entering the network, and analyzingthe property using at least one fuzzy logic algorithm in order to detectthe attack.

Whatever the precise merits, features, and advantages of the prior artis, none of them achieves or fulfills the purposes of the presentinvention.

SUMMARY OF THE INVENTION

The present invention provides for a system and method to detect andmitigate, both distributed and single source IP, HTTP page flood attacks(i.e., HTTP requests floods such as HTTP GET, POST, HEAD etc) that aretypically generated by HTTP bots. The present invention's approach is anovel server-based protection which means that all statistics arecollected per protected server object. Detection of attack/anomaly ismade according to multiple HTTP traffic characteristics includingrate-based and rate-invariant parameters in both traffic directions.Prevention is done according to more in-depth HTTP traffic parametersthat are analyzed once traffic anomaly is detected. This protectionsystem is based on anomaly detection engine that checks correspondenceof current (real-time) traffic parameters to its history learned inregular (non-attack) system state. This protection system includes anadaptive mechanism that tunes the sensitivity of the anomaly detection(or decision) engine according to the adapted normal traffic behavior.The decision engine is based on a combination between fuzzy logicinference systems and statistical thresholds. A “trap buffers” mechanismis responsible for characterizing the attack in a way that allows anaccurate mitigation according to the source IP address(es) of the attackand the HTTP requests that are used as part of the attack.

The present invention provides for a server-based architecture providingHTTP flood protection comprising: a statistics module computing aplurality of real-time statistical parameters, said plurality ofreal-time statistical parameters comprising at least one rate-basedparameter or at least one rate-invariant parameter and one rate-basedparameter; a learning module computing normal base line values; ananomaly detection engine comprising an embedded correlation rules and adegree of anomaly generator generating a degree of anomaly (DoA) basedon said received plurality of real-time statistical parameters and saidplurality of normal base line values; at least one source IP trapbuffer, said at least one source IP trap buffer detecting abnormalrepetitions (frequency) of HTTP request URI's per source IP address andprotected Web server; and at least one HTTP request size trap buffer,said at least one HTTP request trap buffer detecting abnormalrepetitions (frequency) of HTTP request URI's per protected Web serveronly (i.e., without source IP addresses), wherein the generated degreeof anomaly indicates a HTTP flood attack, and the decision enginecommunicates with said source IP trap buffer and the HTTP request sizetrap buffer to characterize the anomaly. The HTTP request size trapbuffer is used for the case in which the attack is distributed amongmany source IP addresses (e.g., more than 1000). In such a case it willnot be possible to analyze the traffic according to source IP address(for memory and CPU resources scalability reasons). Therefore only HTTPrequest URI will be characterized.

The present invention also provides for an anomaly detection enginecomprising: an interface to receive at least the following: a pluralityof real-time statistical parameters or a plurality of normal base linevalues, said plurality of real-time statistical parameters comprising atleast one rate-based parameter or at least one rate-invariant parameterand one rate-based parameter; embedded correlation rules; a degree ofanomaly generator generating a degree of anomaly (DoA) based on saidreceived plurality of real-time statistical parameters and saidplurality of normal base line values; and when said generated degree ofanomaly indicates a HTTP flood attack, said decision engine communicateswith at least one buffer to characterize the anomaly.

In one embodiment, the normal base line values are generated by alearning mechanism comprising of any of the following: day-timedifferential averaging or continuous IIR filtering/continuous averaging.It should be noted that the protection system learns according to bothlearning strategies. The anomaly detection engine uses only one of themaccording to manual configuration or automatically through an automaticlearning strategy characterization mechanism.

In one embodiment, the normal base line values are dynamically updatedover a predetermined time period so that said anomaly detection enginetunes its detection sensitivity based on said updated normal base linevalues.

In one embodiment, the learning mechanism is dynamically picked fromsaid day-time differential averaging or said continuous IIRfiltering/continuous averaging based on a behavior of a networkprotected environment. It should be noted that the anomaly detectionengine always starts (tuned) with the continuous averaging baselines andafter a few weeks a decision if to move into differential method or not(which requires more statistical data), is taken place.

In one embodiment, the anomaly detection engine is implemented in serveraccessible over a network, wherein the network is any of, or acombination of, the following: a local area network, a wide areanetwork, the Internet, or a cellular network. In one specificembodiment, the network is the Internet and the server is a web serverand the network attack is a HTTP flood attack.

The real-time statistical HTTP parameters are divided into two groups:rate-based parameters and rate-invariant parameter. Rate-basedparameters are any (but should not be limited to) of, or a combinationof, the following: number of HTTP requests per second, HTTP outboundbandwidth in a time interval and number of HTTP requests per second persource IP address. Rate-invariant parameters are any (but should not belimited to) of, ratio between HTTP request to outbound HTTP bandwidth ina time interval, HTTP request (URL) size distribution, number of HTTPrequest per TCP connection.

The base line parameters are statistical aggregations of any ofreal-time statistical parameters (rate-based or rate-invariant).Continues Learning forms its base line by means of aggregation accordingto a moving time window and, Differential Learning forms WeeklyPeriodical Base Line that aggregates real-time values according tospecific day time and day in week (it involves both direct 24×7histogram approach and Fourier series approach).

The trap buffers (source and size types) are created according tosuspicious lists. There are two lists: source IP list and size list—Thefirst one includes all suspicious source IP addresses (these addressesare determined according to rate parameters as specified below, thesecond list includes all suspicious HTTP request URL sizes that aredetermined according to the normal URL size distribution as specifiedbelow.

Each list is divided into two sub-lists (high suspicious list and lowsuspicious list). Inclusion at each list is determined according tosuspicious and attack thresholds (i.e., LOW=lower than attack thresholdsand higher than suspicious threshold. High=higher than attackthreshold).

In one embodiment, the buffer further comprises: at least one source IPtrap buffer, said at least one source IP trap buffer storing a firstlist of at least one source IP address that was identified based on anabnormal rate of HTTP requests per source IP or based on the abnormalnumber of HTTP requests per TCP connection that was generated by said atleast one source IP address; and at least one HTTP request size trapbuffer, said at least one HTTP request trap buffer storing a second listof at least one suspicious HTTP request size deviating from an adaptedsize distribution base line.

In an extended embodiment, the first list further comprises: a firstsub-list storing highly suspicious sources determined based on afrequency of suspicious occurrences greater than a threshold; and asecond sub-list of storing low suspicious sources determined based on afrequency of suspicious occurrences lower than said threshold.

In one embodiment, the list further comprises: a first sub-list storinghighly suspicious HTTP request sizes determined based on a suspiciousHTTP request size greater than a threshold; and a second sub-liststoring lower suspicious HTTP request sizes determined based on asuspicious HTTP request size lower than a threshold.

After this stage the system analyze the suspicious traffic through thesource IP or size trap buffers. The source trap buffers (the systempreferable option) analyze all HTTP traffic that the suspicious sources(from the suspicious source list) generate toward the protected serverand the size trap buffer analyze all HTTP requests that match the sizesthat exist in the suspicious size list. Both trap buffer types aim tofind specific URLs that are “intensively” repeated in an abnormalfrequency (the source trap buffer is able to associate the source IPaddress to these URLs and the size trap buffer will not).

The rate-based parameter is any of (but should not be limited to) thefollowing: a get request per second counter—Get_C, which is incrementedby one upon classification of any HTTP Get request toward a protectedserver; a post request per second counter—Post_C, which is incrementedby one upon classification of any HTTP Post request toward a protectedserver; an other request per second counter—Other_C, which isincremented by one upon classification of any HTTP request type that isnot Get or Post toward a protected server; an outbound bandwidth persecond counter—OutB_C, which measures the outbound HTTP traffic that isgenerated by a protected server; and a source HTTP request per secondcounter—S_R_C, which is attached to each source IP address that “owns” aHTTP session toward a protected server and wherein the S_R_C counter isincremented by one upon classification of any HTTP Get or Post requesttoward a protected server.

The one rate-invariant parameter is any of (but should not be limitedto) the following: a HTTP request per connection counter—Con_R_C, whichis maintained per TCP connection and wherein the Con_R_C counter isincremented upon classification of any HTTP Get or Post request toward aprotected server; a ratio between outbound bandwidth to HTTPrequests—Out/R_P, which measures an average outbound bandwidth perinbound HTTP Get or Post requests; and a HTTP request short-term sizedistribution table—S_D_T, which measures the current size distributionof HTTP request URL's, and upon classification of a HTTP Get or Postrequests, the HTTP Get or Post request size is being indexed.

In one embodiment, the Out/R_P parameter is calculated as ratio betweenOutB_C and R_C

where, OutB _(—) C=outbound bandwidth and R _(—) C=Get_(—) C+Post_(—) C.

In one embodiment, the HTTP request size index in said S_D_T iscalculated as follows: Request Size Index=INT(Request Size/10), wheresaid Request Size measures the length in bytes of a request-URL path. Inan extended embodiment, the measure of deviation from expected RequestSize is set over a predetermined time period.

In one embodiment, the learning mechanism maintains a long-term sizedistribution table—L_D_T, wherein a measure of deviation from expectedrequest size occurrences is set over a predetermined time period.

In one embodiment, a set of expected values E(i) in calculated from saidL_D_T as follows: E(i)=P(i)×R_C_(n), where R_C_(n) is the total numberof HTTP request (Post and Get) in one second n and P(i) is the observedfrequency of each URL size grade (size grade i) according to L_D_T. Inan extended embodiment, in order to save computational resources, thenormal, suspect and attack edges matrix may be calculated in advance fora few R_C_(n) cases. The normal, suspect and attacks edges are set asfollows:

${{{Normal}\mspace{14mu} {edge}\text{:}\mspace{11mu} {N(i)}} = {E(i)}},{{{Attack}\mspace{14mu} {edge}\text{:}\mspace{11mu} {A(i)}} = \sqrt{l \times \left\lbrack {{E(i)} \times \left( {{E(i)} + {\sqrt{E(i)} \times P}} \right)} \right\rbrack}},{{{Suspect}\mspace{14mu} {edge}\text{:}\mspace{11mu} {S(i)}} = \sqrt{A \times N}},$

where P is a confidence level, l is a configurable detection sensitivitylevel, and N is the number of entries in said L_D_T.

In an extended embodiment, in order to save computational resources, thenormal, suspect and attack edges matrix may be calculated in advance fora few R_C_(n) cases

The system analyses the suspicious sources and sizes (through the trapbuffers) and generate mitigation patterns, i.e., blocking rules thatinclude: a. source IP address(s) and one or more HTTP request URLchecksums associated to them, or b. just HTTP request URL checksums (incase that the size trap buffers were chosen to characterize the detectedanomaly). After the characterization stage the system starts to mitigatethe attack.

In one embodiment, after said characterization of anomaly, any of thefollowing mitigation mechanisms are implemented: rate limit trafficflows which are defined by Source IP(s) and HTTP request URL's checksum(the checksum value represents URL that was found in thecharacterization state) towards a protected server; rate limit trafficflows which are defined by HTTP request URL checksum(s) toward aprotected server; and blocking source IP(s) toward a protected server.In an extended embodiment, the mitigation mechanisms are activatedaccording to any of the following types of feedback: a DoA that isgenerated according to all inbound traffic parameters prior tomitigation and a DoA that is generated according to remaining trafficafter mitigation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the present invention's protection systemarchitecture for HTTP flood protection.

FIG. 2 illustrates a flow chart depicting how the short term sizedistribution table is updated.

FIGS. 3 and 4 illustrate the behavior of the Max S_R_C and Max Con_R_Ccounters, respectively.

FIG. 5 illustrates the LTD table update flow.

FIG. 6 describes the way that 24×7 differential average values arestored and manipulated (not all parameters are specified in below thediagram).

FIG. 7 illustrates the learning phases and conditions to transit fromcontinuous detection sensitivity to 24×7 differential detectionsensitivity (and back).

FIG. 8 illustrates input MF's edges simulation for HTTP request counters(i.e., R_C edges).

FIG. 9 illustrates input MF's edges simulation for the Max sourcerequest counter.

FIG. 10 illustrates the size deviation process.

FIG. 11 illustrates the decision making engine.

FIG. 12 illustrates the HTTP controller state machine transition rulesand the actions that are taken in each state.

FIG. 13 illustrates a flow chart describing the selection flow thattakes place in the characterization state.

FIG. 14 illustrates the DoA persistency test.

FIGS. 15 and 16 illustrate the creation of the source IP list and HTTPrequest size list, respectively.

FIG. 17 illustrates the source buffer.

FIG. 18 illustrates a flow chart depicting the source buffer eventdriven flow.

FIG. 19 illustrates a flow chart depicting the size trap-buffer eventdriven flow.

FIG. 20 illustrates a flow chart depicting source filter list creation.

FIG. 21 illustrates a flow chart depicting size filter list creation.

FIG. 22 illustrates a flow chart depicting the gradual source rate limitmechanism.

FIG. 23 illustrates a flow chart depicting the mitigation feedbackprocess.

FIG. 24 illustrates a flow chart depicting the attack terminationcondition.

FIG. 25 illustrates a flow chart depicting the gradual size rate limitmechanism.

FIG. 26 illustrates a flow chart depicting the attack terminationcondition (size mitigation case).

DESCRIPTION OF THE PREFERRED EMBODIMENTS

While this invention is illustrated and described in a preferredembodiment, the device may be produced in many different configurations,forms and materials. There is depicted in the drawings, and will hereinbe described in detail, a preferred embodiment of the invention, withthe understanding that the present disclosure is to be considered as anexemplification of the principles of the invention and the associatedfunctional specifications for its construction and is not intended tolimit the invention to the embodiment illustrated. Those skilled in theart will envision many other possible variations within the scope of thepresent invention.

The present invention provides for a system and method to detect andmitigate denial of service and distributed DoS HTTP page flood attacksthat can be generated, for example, by HTTP bots. The presentinvention's approach is a novel server-based protection which means thatall statistics are collected per protected server object. Attackmitigation is made per protected server as well.

Detection of attack/anomaly is made according to multiple trafficparameters including rate-based and rate-invariant parameters in bothtraffic directions. Prevention is done according to more in-depth HTTPtraffic parameters that are analyzed once traffic anomaly is detected.

This protection includes a differential (different learning means thatthe statistics are collected and learned per hour and day in a week) andcontinuous adaptive mechanisms that tune the sensitivity of the anomalydetection engine. The decision engine is based on a combination betweenfuzzy logic inference systems and statistical thresholds. “Trap buffers”mechanism is responsible to characterize the attack in a way that allowsan accurate mitigation according to the source IP(s) of the attack andthe HTTP request URL's that are used as part of the attack.

HTTP Flood Scenarios

Possible Attack Scenarios

The HTTP flood protection aims to cover a few possible attack scenariosagainst web servers. All attack scenarios share similar characteristicwhich include a repeated sequence of HTTP request URL's.

These scenarios include:

-   -   a. Attack that uses a repeated HTTP request pattern from the        same source address    -   b. Attack that uses a repeated HTTP request pattern in the same        TCP connection    -   c. Scenarios a or/and b that are generated from multiple source        IP addresses. In this case, it is assumed that the number of        source IP addresses that participate in these attacks is in the        scale of thousands.    -   d. Scenarios a or/and b that are generated from “infinite”        source IP addresses. In this case, mitigation is based only on        the HTTP request that take part in the attack and not the source        IP addresses.

Logical Architecture

FIG. 1 illustrates the present invention's protection systemarchitecture 100 for HTTP flood protection.

Statistics (Real-Time Traffic Characteristics) 102

Real-Time Traffic characteristics 102 includes the classification ofboth inbound 104 and outbound 106 HTTP traffic coming to and from theprotected web server 108. Statistical parameters include both rate-basedand rate-invariant parameters (all per protected server).

The following parameters are used to detect HTTP flood attacks:

-   -   a. Number of HTTP requests per second    -   b. HTTP outbound traffic rate    -   c. Ratio between HTTP request to outbound HTTP traffic rate per        sec    -   d. HTTP request URL size distribution    -   e. Number of HTTP request per TCP connection    -   f. Number of HTTP requests per second per source IP

These statistics are sent every one-second to both learning mechanism110 and to the anomaly detection engine 112.

The aforementioned decision parameters will be further specified laterin the statistics section.

Learning Mechanism—110

Learning mechanism 110 collects and aggregates the real time trafficcharacteristics through averaging, max, percentile procedure functionsand stores the aggregated values (defined as “normal base lines”). Twolearning strategies are included in this protection system:

-   -   Day-Time Differential Averaging (24×7 statistics) that groups        data for aggregation according to 168 possible hours in week.    -   Continuous—Continuous moving window aggregation that groups data        in recent history interval based on IIR filtering averaging.

Both strategies are applied separately to each kind of statisticalparameter. Choice of the proper learning strategy depends on systemhistory and its dynamics, i.e., stabile systems having quite longhistory are likely to exploit the first strategy if it fits the traffic“behavior” toward the protected server; otherwise, the second one ispreferable.

The present invention's system starts to learn the traffic troughcontinuous IIR filtering; however, it has the capability toautomatically choose which strategy is best according to the behavior ofthe protected network environment (i.e., protected servers). Thisdecision process can take 4 or more weeks.

Periodically (typically every one hour), the learning mechanism 110updates the anomaly detection engine 112 with the associated normal baselines. In return, the detection engine 112 tunes its detectionsensitivity.

Anomaly Detection Engine—112

Anomaly detection engine 112 gets every one-second all RT (real-time)statistics parameters. An embedded correlation rules are applied and adegree of anomaly (DoA) is generated.

According to the degree of anomaly, the present invention's systemdecides if an anomaly, that might reflect HTTP flood attack, exists ornot.

When a decision about an anomaly is made, the system stops (freezes) alllearning processes and starts a process which is responsible tocharacterize the anomaly. The attack characterization process isconducted by the source IP “trap buffers” 114 and HTTP request size“trap buffers” 116.

The type of characterization process is selected and initiated accordingto the type of anomaly that is detected (e.g., request size distributionwas violated or not, max number of HTTP requests per source IP addresswas violated or not etc). The selection process is described later on.

Source IP Trap-Buffers—114

Source IP trap buffers 114 are responsible for detecting “abnormal”repeated HTTP request (URL's) pattern or a sequence of repeated patternsthat are generated by any source IP address toward the protected server108. Source trap-buffers 114 are allocated according to the source IPaddresses that are specified in the source IP addresses “suspiciouslist”.

Suspicious (i.e., abnormal high frequency) HTTP request URL patterns,together with their associated source IP addresses, are detected andthen inserted into an “action filter list”. There are two sub lists, onewhich represent highly suspicious sources (the high source group) andtheir associated HTTP requests, and another that includes lesssuspicious sources (low source group). Later on, during the mitigationprocess, these two groups enable a gradual rate-limit process.

The idea behind source trap-buffers 114 is that a legitimate source IPaddress (although the system thought to be suspicious) will not bedetected as an attacker i.e., abnormal HTTP request URL patterns willnot be identified.

Source IP trap-buffers 114 are extracted from a global pool but are usedspecifically for each protected server 108.

Size Trap-Buffers 116

In cases that an anomaly is detected but there is no suspicious IPaddresses (or too many source IP addresses exist), then the sizetrap-buffers are activated to analyze suspicious HTTP request sizes(suspicious sizes are taken from the suspicious size list). Suspiciousrefers to sizes that deviate from the adapted URL size distribution baseline.

These trap buffers 116 aim to detect abnormal repetition frequency ofHTTP request URL's per suspicious HTTP request size. The detected HTTPrequest URL's are inserted into an action filter list that is used lateron to mitigate the attack. As in the case of source trap-buffer, thereare two sub-lists: High size group and Low size group.

Mitigation Mechanisms—118

Mitigation 118 includes a few mechanisms. One mitigation mechanism is anHTTP request rate-limit mechanism that limits the number of HTTP requestin a gradual manner according to a closed-feedback mechanism. Rate-limitis applied and, if it is not sufficient enough to mitigate the attack,then a more “aggressive” rate limit factor is applied.

The idea is not necessarily to block all packets that participate in theHTTP flood but rather to mitigate it to a level in which the protectedweb server isn't in a denial of service risk.

These mitigation mechanisms are activated per one or multiple source IPaddresses associated with the detected HTTP request URL's or per HTTPrequest URL's only. In extreme cases (which will be explained later on)mitigation is activated on all traffic directed to the protected server108.

The decision about switching between the different mitigation mechanismsis done according to the type of detected anomaly and theclosed-feedback mechanism.

Controller—122

Controller 122 is responsible for the synchronization between alldetection, characterization and mitigation mechanisms. Controller 122 isa finite state machine. State transitions are made according to the laststate information, the DoA and state time-outs.

Management—120

Management 120 allows Configuration Logging and Monitoring state of theprotection system per protected server 108. This monitoring is possiblevia graphic representation of all Base Line parameters and real-timetraffic parameters including:

Graphical representation of the protected web server behavior includesthe following details:

Learning Parameters

-   -   a. Normal, max and deviation of HTTP request per second (per        request method.    -   b. Normal, max and deviation of outbound HTTP B/W per second.    -   c. Normal Ratio between HTTP requests to HTTP outbound B/W per        second.    -   d. Normal HTTP request URL size distribution.    -   e. Adapted max HTTP request per source per second.    -   f. Adapted max HTTP requests per connection

RT Parameters

The same RT parameters are presented in a GUI representation as well.

Statistics Module

Real Time Counters

Rate-Based Parameters

The following are the real time counters that the protection systemsupports:

-   -   Get request per second counter—Get_C    -   The Get_C counter is incremented by one upon classification of        any HTTP Get or Post request toward the protected server 108.    -   Post request per second counter—Post_C    -   The Post_C counter is incremented by one upon classification of        any HTTP Post request toward the protected server 108.    -   Other request per second counter—Other_C    -   The Other_C counter is incremented by one upon classification of        any HTTP request type that is not Get or Post toward the        protected server 108.    -   Outbound HTTP bandwidth per second counter—OutB_C [KBytes]    -   The OutB_C counter measures the outbound HTTP traffic that is        generated by the protected server 108.    -   Source http request per second—S_R_C    -   A S_R_C counter is attached to each source IP address that        “owns” a TCP session toward the protected Web server. The S_R_C        counter is incremented by one upon classification of any HTTP        Get or Post request toward the protected server. The aim is to        derive the typical max value of HTTP Post or Get request per        second that a source (any source IP address) sends toward the        protected server.

Rate-Invariant Parameters Per Protected Server:

-   -   HTTP request per connection—Con_R_C    -   This Con_R_C counter is maintained per TCP connection. It is        incremented upon classification of any HTTP Get or Post request        toward the protected server. The behavior of the Con_R_C counter        is similar to the S_R_C. However, it is not reset every one        second (typically it will be reset every 10 seconds or more).        The counter is removed when the TCP connection is closed (or        expired).

Ratio Between Outbound HTTP Bandwidth to Inbound HTTP Requests—Out/R_P

Out/R_P is a rate-invariant (ratio) parameter that measures the averageoutbound HTTP bandwidth per inbound HTTP Get or Post requests persecond.

The variation of Out/R_P parameter can be high (depending on each webapplication). However, during some HTTP flood attack scenarios we expectto see an abnormal ratio that enables the protection system to support abetter decision making process.

It should be noted that the attack scenario that this parameter refersto doesn't necessarily exhibit a very high rate of inbound HTTPrequests. For example, a relativity low rate oh HTTP requests can resultwith significant large amount of HTTP traffic that is sent from the Webserver, thus can cause bandwidth saturation in the outbound trafficdirection).

This Out/R_P ratio parameter is calculated as follows:

${{{Out}/{R\_ P}} = \frac{OutB\_ C}{R\_ C}},$

where, OutB _C=Outbound bandwidth and, in this case, is measured inKbytes per second and, R_C=Get_C+Post_C

The Out/R_P ratio parameter is updated every one second.

HTTP Request Short-Term Size Distribution Table—S_D_T

The S_D_T parameter measures the size distribution of HTTP requestURI's. Size quantization is 10 Bytes.

Upon classification of HTTP Get or Post requests, the request URL isindexed according to its size. The index identifies an entry in a sizedistribution table (short term and long term tables).

The HTTP request size index is calculated as follow:

Request size index=Int(Request size/10)

Where, the request size measures the length (in bytes) of the requestURL path.

Adaptive URL Size Quantization

There is also an option to use a non-uniform URL size quantization thatis based on decomposition of interval (0, infinity) into number of HTTPURL size bins that are expected to be equivalent (i.e., equal in termsof number of HTTP request hits that each bin is expected to be associatewith). The decomposition is based on an initial, learning-only, periodin which the system collects information about URL size distribution andis tuned periodically (every week) in order to be consistent with theprotected server current state. Through this adaptive URL sizequantization, the system can increase the accuracy of the decisionmaking. For example, when using a uniform URL size quantization, andwhen the adapted probability of certain URL size is very low (e.g.,(P(i))=0.0001), then there is a chance that also a very low rate of HTTPrequests that correspond to this URL size will look like an anomalywhile it is actually a “reasonable” legitimate behavior. This case canbe prevented by the adaptive size quantization option.

FIG. 2 illustrates a flow chart depicting how the short term sizedistribution table is updated. It should be noted that:

-   -   1 sec interval—The interval in which a decision is made. Can be        at the range of 1 to 10 seconds.    -   Two types of size distribution tables are supported: short term        (for real-time anomaly detection as specified above) and long        term table for learning, i.e., calculating the URL size        probability.    -   According to the long-term size distribution table (which is        defined later on), a measure of deviation from the expected        request URL size occurrences is set every one second and is sent        to a decision engine.

Learning Mechanisms

Two learning strategies are supported:

-   -   a. Continuous learning    -   b. 24×7 differential learning

Choosing the strategy that best fits the network environment isautomated through the learning mechanism itself.

Continuous Learning Averaging

Continuous learning averaging is based on first order IIR filter. Itmeans that every pre-defined period (typically 1 second) for learnedvalue (lv) the recurrent equation

lv=(1−α)·lv+α·v is applied.   Eq (1):

Where, v is the currently observed value,

-   α represents the continuous learning response. In most cases it is    defined as

$\alpha = {\frac{\Delta \; t}{T}\ln \; \frac{1}{\beta}}$

where T is the learning window size (e.g., day, week, month etc),

-   Δt is the time measurement unit,-   β=0.05 (asymptotical average value after X iterations)

This approach takes into account all previous history of the learnedparameter but the influence of history with delay of T is lesssignificant

This first order IIR continuous learning is applied to the followingstatistics counters:

Get_C, Post_C, Other_C, OutB_C, and their associated peak values.

These counters are incremented every one second. The following is anexample of how the IIR filter applies to R_C parameter:

R _(—) C _(n)=(1−α)·R _(—) C _(n−1) +α·R _(—) C _(new),   Eq(2):

where, R_C=Get_C+Post_C and a represents the continuous learningresponse. Three α values will enable day, week (default) and monthlearning response sensitiveness.

The following alpha values are set to the continuous average functionsunless other alpha is specified:

a. Day learning sensitivity

${\alpha = {{\frac{\Delta \; t}{T}\ln \; \frac{1}{\beta}} = {{\frac{1}{86400}{\ln \left( \frac{1}{0.05} \right)}} = {3.46 \times 10^{- 5}}}}},$

b. 1 week learning sensitivity

α=4.95×10⁻⁶

c. 30 days (1 month) learning sensitivity

α=1.15×10⁻⁶

Peak Values

In addition to the continuous average also a max (peak) parameter iscalculated. The max value is evaluated according to the last hour andevery one hour it goes thorough a max continuous averaging formula(below is an example for calculation of max HTTP request):

R _(—) C _(n) ^(M)=max_(R _(—) C _(new) ^(M) ,α·R _(—) C _(new)^(M)+(1−α)·R _(—) C _(n−1) ^(M))   Eq(3):

It should be noted that n in equation 3 above refers to hour and not tosecond as in the other cases (e.g., equation 2).

Out/R_P

Average of Out/R_P will be calculated every one second as follows:

$\alpha = {{\frac{\Delta \; t}{T}\ln \; \frac{1}{\beta}} = {{\frac{10}{600}{\ln \left( \frac{1}{0.05} \right)}} = 0.0499}}$

To protect Out/R_P base line from extremely big values, this equation isrestricted by 1000.

Max Con_R_C and Max S_R_C

Max S_R_C (Max HTTP request per source)

The Max S_R_C counter represents the max number of HTTP get request thatare associated with any source IP address in one second.

Every one second the max value, among all counters that are associatedwith the protected server, is set. In case that the value is higher thanthe last second's Max S_R _C then a new Max value is set.

FIGS. 3 and 4 illustrate the behavior of the Max S_R_C and Max Con_R_Ccounters, respectively.

Fading Max Counter Behavior

In order to be responsive for a more close in time measurements, every10 seconds the average Max_S_R_C and Max_Con_R_C are calculated asfollow (below is a Max_S_R_C example):

Max_(—) S _(—) R _(—) C _(n)=max_(Max_(—) S _(—) R _(—) C_(new),α·Max_(—) S _(—) R _(—) C _(n 1)+(1−α)·Max_(—) S _(—) R _(—) C_(new))   Eq (5):

In this case alpha represents shorter learning response values such as.

a. 10 minutes

$\alpha = {{\frac{\Delta \; t}{T}\ln \frac{1}{\beta}} = {{\frac{10}{600}{\ln \left( \frac{1}{0.05} \right)}} = 0.0499}}$

b. 30 minutes (default):

α=0.0166

c. 60 minutes (one hour):

α=0.0008

d. 1 day:

α=3.46×10⁻⁵

Long-Term Request Size Distribution Table—LDT

The long term size distribution table is used to calculate the expectedoccurrences number (E(i)) for size grade (i).

FIG. 5 illustrates the LTD table update flow. Every one hour, a set ofprobabilities P(i), where i=1,2, . . . , N, is calculated. The setincludes a probability for each size grade (i.e., size quantization):

$\begin{matrix}{{{P(i)} = \frac{{L\_ S}{\_ C}_{i}}{M}},} & {{Eq}\mspace{14mu} (6)} \\{{Where},} & \; \\{M = {{\sum\limits_{i}\; {{L\_ S}{\_ C}_{i}}} - {{The}\mspace{14mu} {sum}\mspace{14mu} {of}\mspace{14mu} {all}\mspace{14mu} {counters}}}} & \;\end{matrix}$

L_S_C_(i) is a request counter per size index (i) for long termlearning. N is the number of entries in the distribution table.

Counter's Fading Behavior

In order to be more responsive to events that are closer in time (i.e.,remote in time size distribution is less relevant) and in order to avoid(not eliminate) counter's overlapping; each size counter will adapt afading behavior. Fading will be applied according to the followingformula:

L _(—) S _(—) C _(i) =α×L _(—) S _(—) C _(i−1) +S _(—) C _(new)   Eq(7):

Where,

-   S_C_(new) is the number of size hits in one second (of grade i) and    α is calculated this time as follow:

${\alpha = {1 - \frac{\Delta \; t}{T}}};{{\Delta \; t} = {1\mspace{14mu} {{Sec}.}}}$

The three sensitivities are defined as follow:

a. day: T=3600*24

b. week: T=3600*24*7

c. month: (30 days) T=3600*24*30

It should be noted that M, which is the sum of all counters, iscalculated in the same way (i.e., sum of all fading counters).

In order to have a robust probability measure, probability calculationwill take place every one hour only if M reaches a certain value(between 1000 to 10000 with a default of 5000). If not, then the hourlyprobability update will be skipped.

Counters Normalization

To avoid overflow in counting there is special renormalizationmechanism: In case that a counter, in the LDT, reaches some limit(typically 10⁹,) all counters will be divided by some factor (typically1000).

Continuous Normal Base Lines Summary

Continuous learning parameters table shown in Table 1 below summarizesthe continuous learning section.

Initial Learning Values

Initial learning values are set according to the one of the followingthree options:

-   -   a. Adaptive steps—In case of IIR filters, in the first n^(th)        steps (seconds) alpha are set in a way that result in linear        average expression (alpha=i−1/i; if i<=n)    -   b. Initial rate values are defined through configuration.        Initial peak values are set as well.    -   c. The system is in a learning mode for a pre-defined period or        number of events.    -   During this period there is no detection and prevention (only        monitoring).

24×7 Differential Learning Averaging

In cases of stable periodic traffic behavior, a 24×7 differentiallearning can improve the accuracy of detection.

Differential learning keeps a set of average values for each hour in aday and day in a week. Historical information (one hour average values)of 1 week is maintained. By using IIR continuous averaging, historicalinformation of up to 24 weeks (˜6 months) can have contribution to thedeferential learning base lines. In order to decide if differentiallearning is beneficial, i.e., fits the protected web server behavior, aconvergence parameter (R) is calculated.

If this parameter comply with pre-defined conditions, the systemautomatically tunes the sensitivity of the anomaly detection engineaccording to the 24×7 differential learning base lines.

The anomaly detection engine always starts to work based on thecontinuous learning mechanism. Switching to 24×7 detection sensitivityis possible only after a pre-defined period (default: four weeks) and ifthe convergence parameter (R) meets the pre-defined conditions.

For cases in which the system has to switch back from differentialdetection sensitivity to continuous detection sensitivity, thecontinuous learning is maintained in the background.

Logical Structure of 24×7 Differential Base Lines

The following base line parameters are learned according to differentiallearning mechanism:

a. R_C

b. Other_C

c. OutB_C

The other parameters are averaged according to continues learningmechanism only.

FIG. 6 describes the way that 24×7 differential average values arestored and manipulated (not all parameters are specified in below thediagram). In FIG. 6:

-   -   1. Y_(mn)≡Linear average of one second counter, where n        (n=1,2,3, . . . N) represents day in a week and m (m=1,2,3, . .        . ,M) represents hour in a day. This is a “stand alone” value        that represents a certain hour in a day.    -   2. Ŷ_(m,N)≡Linear average of hour m over one week (for future        use).    -   3. Ŷ_(M,n)≡Linear average of day n over all hours in the day        (for future use).    -   4. Ŷ_(m,n)≡Exponential average of hour m in day n over “all”        weeks. This average function includes adaptive steps for the        first four weeks. The average is calculated every one hour given        the new linear average Y_(mn) and last average Ŷ_(m,n) ^(old)        so,

Ŷ _(m,n) =α×Y _(mn)+(1−α)×Ŷ _(mn) ^(old)   Eq (8)

-   -   5. {circumflex over (Ŷ)}_(m,n)≡Exponential average of day n over        “all” past weeks. This average function includes adaptive steps        for the first four weeks (for future use).    -   6. {circumflex over (Ŷ)}_(m,N)≡Exponential average of hour m        over “all” past weeks. This average function includes adaptive        steps for the first four weeks (for future use).        24×7 learning response values are set as follows:

a. 4 weeks (special case for “big” alpha values):

${{\alpha = {1 - \beta^{\frac{\Delta \; t}{T}}}};} = {{1 - 0.05^{\frac{1}{4}}} = 0.52}$

-   -   where, β=0.05 (asymptotical value after X iterations)

b. 12 weeks (˜3 months—special case for “big” alpha values):

${{\alpha = {1 - \beta^{\frac{\Delta \; t}{T}}}};} = {{1 - 0.05^{\frac{1}{12}}} = 0.22}$

c. 24 weeks (˜6 months):

$\alpha = {{\frac{\Delta \; t}{T}\ln \frac{1}{\beta}} = {{\frac{1}{24}{\ln \left( \frac{1}{0.05} \right)}} = 0.125}}$

Parameters that are not specified in FIG. 6 above include:

-   -   1. Y_(m,n) ^(M)≡Max value (peak) of events for each hour m in        day n. This counter will follow the same max continuous        averaging formula as was specified in (Eq 5). The max average        will be Ŷ_(m,n) ^(M).    -   2. R—Periodic stability measure set of parameters. R is        calculated according to the information of the last 4 weeks and        is a measure of “stability” of series Y_(m,n) ^(i), where i        represents a week number (i=1, . . . 4) and the combination of        m, n represents hour m in a day n.

So,

$\begin{matrix}{R_{mn} = {\sum\limits_{i = 1}^{3}\; {\frac{{Y_{mn}^{i} - Y_{mn}^{i - 1}}}{\min \left( {Y_{mn}^{i},Y_{mn}^{i - 1}} \right)}.}}} & {{Eq}\mspace{14mu} (9)}\end{matrix}$

The above equation represents the “stability” parameter for each hour min a day n. The stability condition is defined as R_(mn)≦1. In case allseries (i.e., all hours) comply with this condition over the last 4weeks, then 24×7 differential learning sensitivity is applied. Conditionflow is defined in FIG. 7.

It should be noted that:

-   -   Initial max/peak values definitions are part of the initial        learning value options (see above).    -   There is an option to add “sub” convergence parameters that will        measure the stability of one day over weeks and one hour over a        few days and weeks and use it in order to update the normal        baselines in a more “customized” manner, e.g., normal base line        per hour or day throughout all weeks.    -   An option to apply additional analysis methods such as Fourier        Transform can be valuable in order to detect periodic behavior        patterns and use them in order to set a proper baselines        strategy (e.g., detect periods only in certain days and hours        etc).    -   Linear average is more accurate in cases of small number of        measurements; therefore, an average of one hour and one day over        one week always use a linear averaging method.

Initial values:

-   -   Parameters that use linear average don't have initial values,        i.e., initial values are set to zero. It should be noted that in        any case, the system will not start with the 24×7 differential        learning strategy (i.e., there is time to collect data).    -   As mentioned above, parameters that use IIR filters use adaptive        steps for the first n^(th) iterations (default of 4 iterations)

Choosing the Learning Strategy

FIG. 7 illustrates the learning phases and conditions to transit fromcontinuous detection sensitivity to 24×7 differential detectionsensitivity (and back).

The initial type of detection sensitivity works according to thecontinuous learning base lines. After the first four weeks the systemcalculates, in step 702, a “stability parameter” (R) that measures if24×7 differential detection sensitivity fits the protected web server ornot. If not, then the system, in step 704, continues to use thecontinuous base lines and, in step 706, re-calculates R every one week(four weeks cyclic calculation). If R meets the stability condition,then the system, in step 708, switches its detection sensitivity to 24×7differential base-lines. The system, in step 706, continues to check Revery one week and act according to the stability condition. In anycase, both 24×7 and continuous learning mechanisms continue to work allthe time except for the time in which anomaly was detected —this will bespecified in the system controller section.

Decision Engine

Major part of the decision engine is based on Fuzzy Logic InferenceSystem (FIS). Every one second, the FIS collects all RT rate-based andrate-invariant parameters and generates a degree of anomaly accordingly.The FIS is adaptive which means that input member ship functions (MF's)are created every one hour according to continuous or 24×7 differentialbase lines.

Rate-Based Parameters Input MF's

This section specifies the way that the membership function edges areset. It should be noted that in some cases the decision will be based onone parameter only. In these cases, a simple threshold is implemented(this will be specified per each case).

R_C and OutB_C input membership functions

The MF's that represent R_C, and OutB_C parameters are set as follow:

-   -   MF's edges: The following formulas refer to 24×7 differential        base lines but also fit the continuous base as well:

Normal edge: N=Ŷ_(mn)   Eq (10)

Attack edge: A=√{square root over (l×Ŷ_(nm) ^(M) ×Ŷ_(mn))}  Eq (11)

Suspect edge: S=√{square root over (N×A)}  Eq (12)

Above equations follow a measure of variance (√{square root over (N)}),a peak “attractor” (Ŷ_(nm) ^(M)) and a detection sensitivity level (l)factor. The logarithmic nature of these formula aims to insure thatlower average values would result in lower detection sensitivity (i.e.,higher ratio between normal and attack edge).

FIG. 8 illustrates the normal, suspicious and attack decision ranges(edges) that are created by the above formulas 10,11 and 12 above.

Other HTTP request types threshold (Other_C)

Head request will follow simple thresholds. These threshold ranges willbe set according to equations 11, and 12 above.

Max request rate per source—Max S_R_C

Max S_Get_C uses only continuous learning. This parameter will followsimple threshold ranges as well. The associated threshold ranges arecreated as follows:

Normal edge: N=Max_(—) S _(—) R _(—) C _(n)   Eq (12):

Attack edge: Eq (13):

$A = \sqrt{{l \times \begin{bmatrix}{{Max\_ S}{\_ R}{\_ C}_{n} \times \left( {{{Max\_ S}{\_ R}{\_ C}_{n}} +} \right.} \\\left. {\sqrt{{Max\_ S}{\_ R}{\_ C}_{n}} \times P} \right)\end{bmatrix}},}$Suspect edge: S=√{square root over (N×A)},   Eq (14):

where P represents the level of expected peak (default=6), and l(1,2, .. . ,5) is a configurable detection sensitivity.

It should be noted that normal threshold isn't relevant for thedetection.

FIG. 9 illustrates such decision ranges.

Max number of request per connection—Max Con_R_C

Same as for max request rate per source.

Rate-invariant parameters input MF's

Out/R_P—ratio between HTTP request (R_C) to outbound bandwidth (OutB_C)

As specified before the RT formula of the ratio parameter is:

${{{Out}/{R\_ P}} = \frac{OutB\_ C}{R\_ C}},$

where OutB_C=Outbound bandwidth in this case is measured in KBytes persecond.

As specified before, this parameter uses only the continuous learningmechanism.

Fuzzy edges will be created as follow:

Normal edge: N=Out/R _(—) P _(n),   Eq (16):

Attack edge: A=N×l×√{square root over (N)},   Eq (17):

Suspect edge:

$\begin{matrix}{{S = \frac{A + N}{2}},} & {{Eq}\mspace{14mu} (18)}\end{matrix}$

where l (2, . . . ,5) is a detection sensitivity factor (configurable).

Request size distribution—Expected occurrences ranges (long-term sizetable)

As mentioned before:

${{P(i)} = \frac{{S\_ C}_{i}}{M}},$

where, C_(i) is a request counter per size index (i) and

$M = {\sum\limits_{i}\; {S\_ C}_{i}}$

is the total number of HTTP Get and Post requests.

A set of expected values E(i) in one second* is calculated as follow:

E(i)=P(i)×R _(—) C _(n),   Eq (19):

where R_C_(n) is the total number of HTTP request (Post and Get) in onesecond*.

*The observation might take a few second before a decision can takeplace (in order to have enough measurements).

In order to save computation resources the normal, suspect and attackedges matrix may be calculated in advance for a few R_C_(n) cases.

The threshold per size grade (i) is set as follows:

${{{Normal}\mspace{14mu} {edge}\text{:}\mspace{11mu} {Eq}\; (20)\text{:}\mspace{11mu} {N(i)}} = {E(i)}},{{{Attack}\mspace{14mu} {edge}\text{:}\mspace{11mu} {Eq}\; (21)\text{:}\mspace{20mu} {A(i)}} = \sqrt{l \times \left\lbrack {{E(i)} \times \left( {{E(i)} + {\sqrt{E(i)} \times P}} \right)} \right\rbrack}},{{{Suspect}\mspace{14mu} {edge}\text{:}\mspace{11mu} {Eq}\; (22)\text{:}\mspace{20mu} {S(i)}} = \sqrt{A \times N}},$

where P is a confidence level (default=9) and l is a detectionsensitivity level (1,2, . . . ,5).

FIG. 10 illustrates the size deviation process 1000. In step 1002, thesize counter S_C_(i) that deviates the most (from normal) identified andE(i) is selected accordingly (i.e., according to size grade i). In step1004, the MF's analytical parameters are updated according to E(i). Theanalytical fuzzy membership function receives the associated fuzzy edgesparameters (N(i), S(i), A(i)) and sets the fuzzy shapes accordingly inreal time.

MF Analytical Definitions:

Normal  MF  function${{S(i)} \geq X > {N(i)}}->\left\{ {Y = {{{- \left( \frac{1}{{S(i)} - {N(i)}} \right)} \cdot X} + \frac{S(i)}{{S(i)} - {N(i)}}}} \right\}$N(i) ≥ X− > {Y = 1} Y = DoF^(N) Suspect  MF  function${{S(i)} \geq X \geq {N(i)}}->\left\{ {Y = {{\frac{1}{{S(i)} - {N(i)}} \cdot X} - \left( \frac{N(i)}{{S(i)} - {N(i)}} \right)}} \right\}$${{A(i)} \geq X > {S(i)}}->\left\{ {Y = {{{- \left( \frac{1}{{A(i)} - {S(i)}} \right)} \cdot X} + \frac{A(i)}{{A(i)} - {S(i)}}}} \right\}$Y = DoF^(S) Attack  MF  function${{A(i)} > X \geq {S(i)}}->\left\{ {Y = {{\frac{1}{{A(i)} - {S(i)}} \cdot X} - \left( \frac{S(i)}{{A(i)} - {S(i)}} \right)}} \right\}$X ≥ A(i)− > {Y = 1} Y = DoF^(A)

In step 1006, S_C_(i) is used as an input into the MF.

Decision Making Engine

FIG. 11 illustrates the decision making engine. There are four decisionengines 1102, 1104, 1106, and 1108. Each of the decision enginesgenerates DoA in an independent way (every one second). These enginesare updated every one hour (MF's update and thresholds update) accordingto the adapted normal base lines (24×7 or continuous base lines). Engine3 (FIS1) 1106 Max (deviation) MF is updated upon every size input (i.e.,S_C_(i), E(i)) and the R_C MF is updated every one hour according to theR_C normal base-line.

Degree of Attack Triggers

DoA values are generated per each decision engine in the following way:

DoA Ranges

Engine 1—Other_C DoA ranges

Degree of attack is calculated as follows:

-   -   Normal=Other_C<Other_C_(suspect),    -   Suspect=Other_C_(attack)>Other_C≧Other_C_(suspect),    -   Attack=Other_C≧Other_C_(attack),        where Normal=2 (not relevant for the decision it self);        Suspect=6 and Attack=10

DoA Characteristic: this DoA mechanism represents simple HTTP requestflood that utilizes requests that are not Get or Post. It should benoted that the impact of this attack is usually low. Still, frequentHTTP requests (HEAD, PUT etc) can represent web intelligence gatheringactivities that need to be prevented.

Engine 2—Per source request rate and per connection # of requeststhresholds

Degree of attack is calculated as follows:

-   -   Normal=Max_S_R_C<Max_S_R_C_(suspect),    -   Suspect=Max_S_R_C_(attack1)>Max_S_R_C≧Max_S_R_C_(suspect),    -   Attack=Max_S_R_C≧Max_S_R_C_(attack),        where Normal=2; Suspect=6 and Attack=10.

It should be noted that the same goes for Max_Con_R_C. In this case, themaximum among these two counters determines the DoA for a decision.

DoA Characteristic: this DoA mechanism represents an inbound HTTP pageflood which significantly raises the request rate per source IP or thetotal number of request per connection toward the protected server.

Engine 3 (FIS 1)—R_C/S_Ci

DoA ranges are defined as follows:

-   -   Normal=2-4    -   Suspect=5-7    -   Attack=8-10

DoA Characteristic: this DoA mechanism represents an inbound HTTP pageflood which significantly raises the request rate toward the protectedserver (above the normal base lines).

Engine 4 (FIS 2)—OutB_C/OutB/R_P

DoA ranges are the same as for FIS 1.

DoA Characteristic: this DoA mechanism represents an inbound HTTP pageflood which significantly raises the outbound traffic rate from theprotected server—note that this attack isn't necessarily high raterequest attack.

System Controller

The controller is responsible for the synchronization between thedecision engine outputs and the mitigation methods.

The controller is a finite state machine that includes four majorstates:

a. Normal state (0)

b. Characterization state (1)

c. Mitigation state (2)

d. Suspicious activities state (3)

The controller is an event driven and time driven mechanism. FIG. 12illustrates the transition rules and the actions that are taken in eachstate.

Normal state (0)

In “peace time” the system stays in the normal state. During this statesthat system continuously collect HTTP traffic statistics and update thenormal base lines accordingly.

Characterization state (1)

Transition from normal state to characterization state occurs when oneor more of the anomaly triggers is activated (1 second intervaldecision). During the operation of this state, if the DoA values arebelow 8 (“Attack range), a transition back to the normal state isoccurred. During this state no learning is taken place, i.e., base linesare “freezed”. This state comprises two internal processes (written intheir actual order):

a. Selection of attack characterization method; and

b. Creation of action filter list.

Characterization Method Selection

The first process includes a decision making process about the type ofanomaly that was detected and about the attack characterization methodthat need to be used in order to create action filters. This decision ismade according to pre-defined rules that will be specified later on inthe paper. This phase lasts between 2-4 seconds (default 4 seconds) inorder to check the consistency of the anomalous parameters beforedeciding about the best attack characterization method.

Creation of Action Filters List

According to the selection process, the system activates the appropriatetrapping mechanism that creates an action filters list. The list isordered according to risk/impact analysis. The list includes suspicioussource IP addresses, HTTP request sizes and request checksum(s). Thisprocess lasts between 4-10 seconds (default 6 sec) in which the actionfilter list is getting filled. When this phase is over (expired), atransition to state 2 or 3 is taken place.

Suspicious Activities State (3)

A transition to state 3 occurs in case that the action filter list isempty, i.e., the system couldn't find any potential countermeasure forthe detected anomaly, or if the system has failed to mitigate the attackafter a few iterations (this process will be explained later on). Duringthis state no learning is taken place, i.e., base lines are “freezed”.

During this state no actions are taken place and only anomaly statisticswill be presented for the user.

A transition back to state 1 occurs in any case after an expirationperiod (default 120 seconds).

A transition to normal state (0) occurs if DoA below 8 is indicated. Itshould be noted that this state may include an optional blocking/ratelimit mechanism. This mechanism rate limits the HTTP requests that aregenerated by specific source IP addresses. These sources can be selectedaccording to the S_R_C or Con_R_C statistics.

Mitigation state (2)

No learning takes place in this state.

Mitigation actions are activated in this state. The mitigation actionsare selected according to the action filter list that was created instate 1.

Mitigation options include:

-   -   a. Rate limit the traffic flows which are defined by Source        IP(s)+HTTP request(s)/URL(s) checksum toward the protected        server    -   b. Rate limit the traffic flows which are defined by HTTP        request checksum(s)/URL(s) toward the protected server    -   c. Rate limit all traffic that is generated from specific source        IP address(es) toward the protected servers

These mitigation options are activated according to a closed feedbackoperations. There are two types of feedback:

-   -   a. Full DoA—DoA that is generated according to the all inbound        traffic parameters, i.e., before the mitigation actions.    -   b. Filtered DoA—DoA that is generated according to the remaining        traffic after mitigation action took place. This feedback type        is more relevant when rate limit mitigation actions are taken        place (i.e., during source IP full blocking, the filtered DoA        isn't relevant for positive feedback purposes). Positive        feedback means effective blocking. Negative feedback means        non-effective blocking.

Generally, the mitigation action works gradually (from “gentle” to“rough”) which means that in case of negative feedback the rate limitfactor is adjusted (i.e., more drastic blocking measures are created).If all rate limit steps are implemented and still negative feedback isindicated, a transition back to state 1 is occurred.

After X continuous cases like this (X is between 2 and 4), the systemwill transit to state 3 (suspicious activities state).

State Expiration Period

A transition back to state 1 occurs when the state time-out is expired(expiration period is between 60 and 600 seconds. Default: 600). Thistransition is involved with removal of all filters. The reason for thistransition is to allow the system to get-out from scenarios in which it“thinks” that the mitigation process works effectively while it reallyblocks legitimate traffic.

Transition to Normal State (Attack Termination)

In case that the system uses rate limiting mitigation actions, thesystem will transit to normal state (i.e., attack termination) ifpositive feedback is indicated according to both Full and Filtered DoAvalues. In case the system blocks traffic (i.e., blocks all HTTP trafficthat s generated from a source address ), there is no sense in measuringthe filtered DoA for positive feedback decision (i.e., only negativefeedback can be indicated in case that the wrong source IP is blocked).In this case, a transition to normal state will be decided according toa special feedback process—this will be defined later on.

The following sections will define, in a more granular manner, thedecision and actions that are taken place in each one of the states:

State 1—Attack Characterization

Characterization Method Selection

According to the type of triggers that are responsible for transition tostate 1, the system decides which attack characterization method willbest fit the case. Table 2 below analyzes the relationship betweenpossible HTTP flood attack characteristics, the decision engines and thecharacterization method:

TABLE 1 Attack characteristics options Suspicious source Suspiciousrequest Characterization # Trigger case addresses(es) size(s) methoddecision Comment 1 Other_C trigger N/R N/R This case alone This casewill be (Engine 1) doesn't require attack followed by, in anycharacterization case, “Other” request process blocking 2Source/connection Exist (always) N/R Source IP trap buffers Detectsrepeated thresholds HTTP request (Engine 2) pattern(s) per suspicioussource(s). 3.1 FIS 1 - R_C/ Exist Exist (always) Source IP trap buffersDetect repeated HTTP S_Ci request pattern(s) per (Engine 3) suspicioussource(s) 3.2 FIS 1 - R_C/ Not exist Request size trap Detect repeatedHTTP S_Ci buffers request pattern(s) per (Engine 3) suspicious requestsize(s) 4.1 FIS 2 - Exist Not Exist Source IP trap buffers Detectrepeated HTTP OutB_C/ request pattern(s) per OutB/R_P suspicioussource(s) (Engine 4) that cause uplink B/W consumption. 4.2 FIS 2 - Notexist Exist Request size trap Detect repeated HTTP OutB_C/ buffersrequest pattern(s) per OutB/R_P suspicious request size (Engine 4) (s)4.3 FIS 2 - Not exist Not exist No characterization Anomaly detectionOutB_C/ only (suspicious state). OutB/R_P (Engine 4)

Simultaneous Triggers

The characterization method is chosen after a consistency check that isdone in state 1, i.e., only persistent triggers (high DoA that isconsistent) will take part at the decision. Simultaneous triggers willbe handled as follows:

-   -   a. Persistent Other_C trigger will always be followed by        blocking of any HTTP request which is not Get or Post regardless        to the other triggers as mentioned in the table above    -   b. In case both FIS1 (engine 3) and FIS2 (engine 4) generate        high (and persistent) DoA then characterization process will        treat it as FIS1 (engine 3) trigger type.

FIG. 13 illustrates a flow chart describing the selection flow thattakes place in the characterization state. It should be noted that thetransition to state 1 from state 0 is done according to the max DoAvalue among all the DoA's that are generated by the four engines.

Persistency Test—1302

The persistency test in state 1 includes a simple Full DoA countercondition. The condition is defined as shown in FIG. 14 per each one ofthe four anomaly engines. In FIG. 14, P_A is a Persistency Attackcounter.

Creating Source and Size Suspect Lists—1304

During the first 4 seconds of the persistency test in state 1 the systemcollects all suspicious sources and HTTP request (URL's) sizes andcreates the following two lists:

a. Source IP suspect list

b. HTTP Request size suspect list

Each list contains two sub-groups: High group and Low group. FIGS. 15and 16 illustrate the flow defining how these two suspect lists arecreated. FIG. 15's flow refers to both source request counter (onesecond counter) and connection request counter (i.e., number of HTTPrequest per TCP connection).

Suspect List Restriction

A condition on the number of suspicious sources and sizes is enforced.The condition will be that above 1000 (range can be between 1000 to5000) suspicious source IP addresses, the system will select the sizetrap buffer attack characterization method. In case that the number ofsuspect request sizes (URLs) is above 20 (range can be between 5-100),then the system will transit to the suspect state (state 3).

Source Action Filter List Restriction

Per source IP, a limitation of no more than 10 requests URL filters willbe implemented in the source action filter list.

Motivation: Too many suspicious URL sizes or source IP addresses meanshigher probability of false positive decisions.

Action Filter List Creation (State 1)

According to the information that was collected in the previous processin state 1 (i.e., content of the suspect lists), an action filter listis created according to an internal attack characterization process.Attack characterization is made trough two types of “trap” buffermechanisms:

a. Source trap buffer

b. Size trap buffer

This characterization process lasts between 4-10 seconds (default 6)seconds.

Source Trap Buffer

Source trap buffers can be created for each one of the source IPaddresses that are in the source suspect list. Each buffer is N cellslong (default of 5 cells an option to extend it up to 10 sells isimplemented). Each cell contains:

a. One checksum value S that represents a specific HTTP request-URI.

b. HTTP Request counter C_(s)

c. HTTP request threshold Th_(s) (it is a dynamic threshold)

Each cell represents a different HTTP request URL that have been made tothe protected web server from the associated source IP address and thecounter represents the number of repeated same URL requests. When thecounter breaches the request threshold Th_(s), the checksum valuetogether with the actual request-URL is added to an action filter list.In the case of source trap-buffer, the request-URL (or a few URL's) isadded to the action filter list together with the associated source IPaddress.

The trap buffer is a time driven cyclic buffer, meaning that HTTPrequest(s) are shifted up in the buffer every ΔT second (together withtheir associated counter and dynamic threshold). In case that that a newrequest arrives to a full buffer (i.e., all buffers are alreadyoccupied), it is ignored.

The threshold Th in each cell is tuned/updated upon every shift (i.e.,ΔT) until it arrives to the N^(th) cell. The “oldest” request stays inthe last cell of the buffer until a new not existing request is arrived(this is important in order to detect relatively low rate HTTP requestURL flood).

FIG. 17 illustrates the cyclic buffer structure of the source buffer. InFIG. 17:

$\begin{matrix}{{Th}_{s} = {\sqrt{\left\lbrack {{{\overset{\_}{E}}_{s}(i)} \times \left( {{{\overset{\_}{E}}_{s}(i)} + {\sqrt{{\overset{\_}{E}}_{s}(i)} \times P}} \right)} \right\rbrack}..}} & {{Eq}\mspace{25mu} 23} \\{{{\overset{\_}{E}}_{s}(i)} = {{{{P_{s}(i)} \times R_{rps} \times \Delta \; T \times {Cell}} - n}..}} & {{Eq}\mspace{20mu} 2\; 4}\end{matrix}$

S≡Checksum value of the HTTP request URL.

C_(s)≡Number of checksum S hits.

P_(s)(i)≡The adapted probability of the size (i.e., size grade) of theassociated HTTP request URL.

P≡Confidence level.

R_(rps)≡The HTTP request rate (request per second) origin at the sourceaddress.

This rate is measured in the first second after the source trap bufferis created and is used for the rest of the trap-buffer lifecycle. Anoption to update this parameter every one second will be implemented aswell (in order to have a better accuracy in cases of dynamic rateattacks).

ΔT≡Cell shift time is 0.5 sec (range can be between 0.1 to 10 sec).

cell−n≡Current cell number (n=1, . . . , N) and N≡buffer length (N is aconfigurable value).

FIG. 18 illustrates a flow defining the source trap buffer event drivenupdate procedures. It should be noted that:

-   -   Source trap buffer is created upon classification of the first        request from a “suspicious” source IP address, i.e., an address        that exists in the low or high sub-lists of the suspicious        source list.    -   Trap-buffers are terminated upon transition to mitigation state.    -   This trap buffer should handle problematic scenarios such as        HTTP proxy low rate HTTP floods, mix rate attacks, attacks with        decoys, and high probability request-URL flood.

Size Trap Buffer

In case that an anomaly was detected but only suspicious HTTP requestURL sizes were detected (or the number of suspicious sources have passedthe pre-defined restriction), then the system activates the size trapbuffer. This is a time driven cyclic buffer that aims to detect repeatedHTTP requests per request-URL size (i.e., size grade).

This buffer mechanism is identical to the source IP trap buffersmechanism. The output of each size trap-buffer is one or a set of HTTPrequest checksum value(s), the actual request-URL strings, and theassociated exact string size.

In this case, each trap buffer represents one URL size grade that wasdeviated from the normal expected occurrences value (E(i)). Therefore,Ē(i) is set per each trap buffer. The thresholds are set as follows:

$\begin{matrix}{{{Th}_{size} = \sqrt{\left\lbrack {{\overset{\_}{E}(i)} \times \left( {{\overset{\_}{E}(i)} + {\sqrt{\overset{\_}{E}(i)} \times P}} \right)} \right\rbrack}},{.{where}}} & {{Eq}\mspace{20mu} 25} \\{{\overset{\_}{E}(i)} = {{{{P(i)} \times R_{rps} \times \Delta \; T \times {Cell}} - n}..}} & {{Eq}\mspace{20mu} 26}\end{matrix}$

-   -   P(i)≡The occurrences probability according to the size (size        grade) of the associated trap-buffer.    -   P≡Confidence level.    -   R_(rps)≡The HTTP request rate (request per second). This rate is        measured in the first second after the size trap buffer is        created and is used for the rest of the analysis operations.        Please note that this rate refers to the overall HTTP request        rate toward the protected server.    -   ΔT≡Cell shift time 0.5 sec (range of 0.1-10 sec).    -   cell−n≡Current cell number (n=1, . . . , N) and N≡buffer length        (configurable between 3-10. default:5).

False negative note: Because the HTTP request size distribution isactually per size range and is not tailored per specific size and alsobecause that different HTTP requests URL's can be of the same size, theprobability is limited in its accuracy. This inaccuracy will can resultin misdetection events (i.e., false negative).

Analyzing probability of each HTTP request URL or request size isn'treally a solution that can scale, therefore the size grade strategy waschosen.

FIG. 19 illustrates the flow describing the event driven updatemechanism.

Action Filter List Creation

The final step in the characterization state (state 1) is to create anaction filter list consisting of source IP addresses and theirassociated HTTP request(s) (i.e., actual request-URL strings) or, listof HTTP request sizes and their associated HTTP request-URL strings—allaccording to the trap buffers' outputs.

FIG. 20 illustrates the flow describing the process in which theseaction filter lists are created. In FIG. 20,

-   -   C_(s) is the counter that represent the number of checksum (S)        hits.    -   High action filter group—The list of source IP addresses that        belong to the high (more “dangerous”) group    -   Low action filter group—The list of source IP addresses that        belong to the low group (less “dangerous”)    -   C is the number of different checksum values (S) that are        associated with the same source IP address.    -   Th_(c)—This threshold controls the number of different checksum        values that can be associated with one source IP address. Above        this threshold, it will be too risky to block/rate-limit the        source and its associated HTTP request. Therefore, the source        will be removed from the action filter list and the suspicious        source list (so it will not be detected again).        The same process goes for the size action filter list creation:

FIG. 21 illustrates size filter lists creation.

Mitigation state (state 2) & feedback operations

In this state all mitigation actions are taken place based on actionfilter lists that were created in state 1. The mitigation process isgradual, i.e., starts “light” and become more “aggressive” if negativefeedback is identified. The mitigation process stops when an end ofattack (attack termination) is identified.

There are four different mitigation methods that can take place in thisstate (some of them simultaneously):

-   -   a. HTTP “other” request types blocking—block HTTP request that        are not Get or Post.    -   b. Source mitigation—Rate limit specific HTTP request URL(s)        that are generated by specific IP addresses. Rate limit is set        according to a predefined rate-limit factor steps.    -   c. Size mitigation—Rate limit specifics HTTP request URL(s). The        rate limit is set according to the adapted E(i).

“Other” Request Blocking

In case that “other” request types flag was set in state 1, during state2 the system will block any request that is not Get or Post.

Identification of attack termination in case of pure “Other” HTTPrequest types flood attack will be according to full DoA values (seebelow how it works in case of source mitigation).

Source Mitigation

In case the source mitigation method was chosen in state 1, upontransition to state 2, the system starts to prevent the HTTP floodattack according to the source action filter lists (High and Low lists).The action filter list includes source IP addresses and their associatedHTTP request-URL(s).

FIG. 22 illustrates the gradual source IP rate limit mechanism.

Rate limit factors—F_(i)

The rate limit applies to all source IP's and their associated HTTPrequest-URL(s) strings. This factor is actually an HTTP request ratelimit. One of two options to set the rate limit factor will availablethrough configuration:

-   -   a. Automatic rate limiting    -   b. Manual rate limiting—manual rate limit will be defined later        on.

In automatic mode the rate limit factor F_(i) will be set as given belowin Table 3:

TABLE 2 Rate limit Rate limit percentage (traffic step (i) F_(i)offload) Comment 1 2 50% Initial factor 2 4 75% Set upon negativemitigation feedback indication 3 F_(f) ¹ = ∞ 100%  Full block ofsuspicious http request URL that are generated by the suspicious sourceIP(s) (In order that this action will be allowed, a full block modeshould be enabled) 4 F_(f) ² = ∞ 100% Any http request URL rate limit(configurable) factor (In order that this action will be allowed, a fullblock mode should be enabled)where, F_(i) is the rate-limit factor that refers to HTTP requesttraffic that is generated from all sources in the block list. Forexample, if the value of this factor is 4 then every 4^(th) HTTPrequest, that matches one of the blocked sources and its associatedchecksum value, will be allowed to pass.

There are four rate limit steps that are taken place according to theclosed-feedback process. The final steps F_(f) represents a “full” (veryaggressive) blocking method and will only be used in case the protectionsystem was configured to work in a “Full block” mode (i.e., “full block”mode is enabled).

Full block mode—F_(f) ² means that all HTTP requests URLs that aregenerated by the suspicious source IP addresses are rate limited. F₄ isa configurable factor—possible values are:

a. 25% (25% http request URL traffic offload)

b. 50%

c. 75%

d. 100% (default—100% means that all traffic from the suspicious sourcesis blocked (including the SYN packets)

The default Th¹ value is 2. In case of “Full block” mode, Th¹ is set to4 (because there are two more feedback iterations that can take placeand mitigate the attack).

Mitigation Feedback Process

FIG. 23 illustrates the mitigation feedback process to identify positiveand negative feedback.

It should be noted that a mitigation negative feedback persistencycounter should be considered (i.e., in case of oscillation no feedbackwill be indicated and the existing status will remain unchanged). Thiscounter will identify if the negative feedback is persistent for between2 to 4 seconds (default 2 sec)

Transition Back to State 1 (Unsuccessful Mitigation)

If mitigation negative feedback is identified after all rate limit stepswhere implemented, the system will remove all sources from the blocklist and a transition back to state 1 will take place. This transitionwill help handle cases in which the attack's characterization ischanging (e.g., new IP address attack the server, new HTTPrequest-URL(s) are used as part of the HTTP flood etc).

After a predefined number of transitions back to state 1 (Th², default3), the system will transit to state 3 (suspect state).

End of Attack Identification

FIG. 24 illustrates the attack termination condition (source mitigationcase). End of attack is identified in state 2 according to the processdescribed in FIG. 24. When the Full DoA stays, persistently, below 8,blocking will be temporarily deactivated. If during this the Full DoAstays below 8 for a pre-defined period (default 30 seconds) then thesystem will transit to state 0, i.e., attack termination.

The motivation is to check if the attackers adapt the rate condition(through their TCP's flow control mechanism—this is relevant for a moresophisticated HTTP flood tools), thus will prevent frequent re-openingof continuous HTTP flood attack. If during the 30 seconds the systemidentifies an attack (i.e., Full DoA=>8), the blocking is implementedagain until the termination condition is met again.

Low hit condition—This condition is relevant only when the suspicioussource IP addresses are completely blocked (i.e, final rate limit stepof 100%).

This scenario doesn't allow a good measure of Full DoA because an HTTPconnection cannot be established. Therefore, termination will beexecuted only if the block list is “idle” for a pre-defined period(default: 10 sec). If the low hit rule is met and the Full DoA is stillbelow 8, a transition to state 0 occurs. “Idle” means that no block listhit occurred

It should be noted that the full block mode can be problematic when theattack is generated behind a proxy server (i.e., all connections will beblocked and new, legitimate; connection attempts will maintain a steadyactivity which will not allow terminating the attack). Having said this,proxy's source IP address will probably not be inserted into the blocklist in the first place because the source trap buffer will not detectsuspicious HTTP request checksum value(s) when significant legitimatetraffic is generated from the same source IP address. This problematiccase will usually result in transition to anomaly state (3).

Size Mitigation

In case this method was chosen (in state 1), upon transition to state 2,the system starts to prevent the HTTP flood attack according to the sizeaction filter list.

FIG. 25 illustrates the gradual size rate limit mechanism.

Rate-Limit Factors—F_(i) (Size Mitigation Case)

When the mitigation method is per request URL only (i.e., no suspicioussources were detected or because the number of suspicious sourcesrestriction), a different rate limit factor applies to each specificrequest-URL(s). The rate limit in this case is set according to theadapted URL size probability (occurrences value E(i)), thus the attackis normalized (although the normalization might slow legitimate HTTPrequests that matches the request that is used by the attack itself).

Although the normal expected occurrences value is per size grade and nota specific HTTP request-URL (i.e., the actual normal rate of thespecific request can be lower), we expect that the rate limit processwill be sufficient enough during attacks.

As in the case of source IP mitigation method, also here there are twooptions to set the rate limit factor (configurable):

-   -   a. Automatic rate limiting    -   b. Manually rate limiting—(defined later on)

Automatic Mitigation Method

In automatic mode the rate limit factor F_(i) will be set as follow:

Initial rate limit factor−F ₁ =Ē(i)×ΔT,

where

-   -   ΔT≡Is the rate limit time unit (default: 1 sec)    -   Ē(i)≡Represents the expected number of HTTP request-URL of size        i (grade i ).    -   Ē(i) is the same value that was calculated in the size        trapbuffer (i.e., the one that was used for setting the        threshold in the first second—see Eq. 26).

The rate limit steps will be set according to Table 4 below:

TABLE 3 Rate limit step (i) F_(i) Comment 1 F₁ = Ē(i) × ΔT Initialfactor 2 F₂ = F₁ × 0.5 Set upon negative mitigation feedback indication3 F₃ = F₁ × 0.1 3^(rd) and more aggressive rate limit factor 4 F_(f) = 0When 100% is set, all suspicious (configurable) request-URLs will beblocked toward the protected server.

In Table 4, F_(i) is the rate limit factor that is associated withspecific HTTP request-URL.

There are four rate limit steps that are taken place according to thefeedback process. The final step F_(f) is a full HTTP request-URL blockand will only be used in case the protection system was configured towork in a “Full block” mode (not the default mode). The default Th¹value is 3. In case that “Full block” mode is enabled, Th¹ is set to 4.

Full block mode—Last rate limit step Th¹ can be configured with thefollowing values:

a. 25% (25% http request URL traffic offload)

b. 50%

c. 75%

d. 100% (default—100%, means that all suspicious request URL's areblocked towards the target HTTP server.

Mitigation Feedback Process

Mitigation positive and negative feedbacks are handled in the same wayas in the case of source mitigation.

Transition Back to State 1 (Unsuccessful Mitigation)

If mitigation negative feedback is identified after that all rate limitsteps where implemented, the system will remove all request-URL(s) fromthe block list and a transition back to state 1 will take place.

This transition will help handle cases in which the attack'scharacterization is changing (e.g., new HTTP request-URLs are used inthe HTTP flood). After a predefined number (Th²) of transitions fromstate 2 to 1, the system will transit to state 3 (suspect state)—This isthe same behavior as in the case of source mitigation.

Attack Termination Condition (Size Mitigation Case)

FIG. 24 illustrates the end of attack condition (size mitigation case).Attack termination is identified in state 2 by FIG. 26. When the FullDoA stays, persistently, below 8, blocking will be temporarilysuspended. If during this period the Full DoA is below 8, the systemwill transit to state 0 (i.e., attack will be terminated). Themotivation behind this process is the same as for the source mitigation.In this case the low hit condition isn't relevant because TCPconnections are allowed to be established (also when a full block modeis enabled).

Blocking Filter Format:

The blocking rules that were defined in the mitigation process areprovided below in Table 4:

TABLE 4 Blocking filter options: Max number of values (per # Mitigationfilter options system) 1 {Source IP} [IP] AND Source IP's: 5000{Destination IP} [IP's] AND Checksum's per Source IP: 5 {destinationport} [Port] AND {HTTP request type} [Get or Post] AND {Request-URI}[string(s)] 2 {Destination IP} [IP's] AND Checksum's: 1000 {destinationPort} [Port] AND {HTTP request type} [Get or Post] AND {Request-URI}[string(s)] 3 {Destination IP} [IPs] AND N/R {Destination Port} [Port]AND {request type} [Other: Put, Delete . . . ]

Manual (Non-Adaptive) Attack Triggers Mode

An option to configure manual triggers is implemented. Operation in thismode means that:

-   1. Learning mechanism doesn't influence the detection engines (i.e.,    no thresholds and FIS updates)-   2. FIS 1 (engine 3) and FIS 2 (engine 4) are disabled (i.e., N/R for    decisions)-   3. Instead of decision engines 3 and 4, R_C and OutB_C triggers are    implemented as simple thresholds.-   4. LDT (size distribution) will not have any influence on the    decision during this mode of operation.-   5. Attack triggers are set according to threshold configuration. The    following thresholds per protected server are need to be configured:

a. R_C attack threshold

b. OutB_C attack threshold

c. Other_C attack threshold

d. S_R_C attack threshold

e. Con_R_C attack threshold

Please note that aforementioned thresholds represent attack and notnormal values. Threshold will be ignored in case that it isn'tconfigured, i.e., the system will not trigger an attack according to anon-configured threshold.

-   6. Characterization state (1)—the output of the characterization    state is the mitigation method, i.e., source or size mitigation    (only high source group is relevant in this state). In this mode    trap-buffers cannot be activated which means that filters are not    created.-   7. Mitigation state (2): http rate limit is applied according to the    manual mitigation settings (see 11 below).-   8. No negative feedback operations exist in this operation mode,    i.e., Th¹ is set to one.-   9. Suspect state (3): same operation as in the case of adaptive    decision flow-   10. Attack termination conditions are the same as in the case of    adaptive decision flow. This includes the low hit condition in case    of full block-   11. Manual mitigation settings:

Source Manual Mitigation Rate Limit Mode

In case source mitigation method is activated the following rate limitsettings are relevant Possible manual rate limit factors are:

a. 25%—offload 25% of HTTP requests that are generated by all suspicioussources

b. 50%

c. 75%

d. 100%—100% means that all traffic from the suspicious sources isblocked (including syn packets).

Size manual mitigation rate limit mode (please note that we are usingthe learning in any case in order to find suspect request-URL size)

Possible manual rate limit factors are:

a. 25%—offload 25% of the suspicious request URL's toward the attackedserver

b. 50%

c. 75%

d. 100%—100% means that all HTTP request URL's will be blocked towardthe attacked server during the attack.

Illustration of Decision Ranges

R_C and OutB_C Decision Ranges

FIG. 8 illustrates the normal, suspicious and attack decision ranges(edges) that are created by previously described equations 7, 8 and 9.

The second plot shows the ratio between attack edge and normal edge(810) and between attack edge and suspect edge (812).

Legend:

806—Normal edge

804—Suspect edge

808—Attack edge

802—Peaks (Legitimate peaks)

S_R_C and Con_R_C Decision Ranges

FIG. 9 illustrates the normal, suspicious and attack ranges (samerepresentation as in FIG. 8).

Additionally, the present invention provides for an article ofmanufacture comprising computer readable program code contained withinimplementing one or more modules to implement a method providingadaptive behavioral HTTP flood protection. Furthermore, the presentinvention includes a computer program code-based product, which is astorage medium having program code stored therein which can be used toinstruct a computer to perform any of the methods associated with thepresent invention. The computer storage medium includes any of, but isnot limited to, the following: CD-ROM, DVD, magnetic tape, optical disc,hard drive, floppy disk, ferroelectric memory, flash memory,ferromagnetic memory, optical storage, charge coupled devices, magneticor optical cards, smart cards, EEPROM, EPROM, RAM, ROM, DRAM, SRAM,SDRAM, or any other appropriate static or dynamic memory or data storagedevices.

The present invention also provides for computer readable program codeimplementing: an interface aiding in receiving at least the following: aplurality of real-time statistical parameters or a plurality of normalbase line values, said plurality of real-time statistical parameterscomprising at least one rate-based parameter and rate-invariantparameter; embedded correlation rules; a degree of anomaly generatorgenerating a degree of anomaly (DoA) based on said received plurality ofreal-time statistical parameters and said plurality of normal base linevalues; and when said generated degree of anomaly indicates a networkattack, said decision engine communicates with at least one trap bufferto characterize the anomaly and then to mitigate it trough closedfeedback rate-limit mechanism.

Conclusion

A system and method has been shown in the above embodiments for theeffective implementation of an adaptive behavioral HTTP floodprotection. While various preferred embodiments have been shown anddescribed, it will be understood that there is no intent to limit theinvention by such disclosure, but rather, it is intended to cover allmodifications and alternate constructions falling within the spirit andscope of the invention.

1. A method to detect an anomaly comprising: receiving at least thefollowing: a plurality of real-time statistical parameters and aplurality of normal base line values, said plurality of real-timestatistical parameters comprising at least one rate-based parameter andat least one rate-invariant parameter; maintaining a plurality ofembedded correlation rules; generating a degree of anomaly (DoA) basedon said received plurality of real-time statistical parameters and saidplurality of normal base line values; and when said generated degree ofanomaly indicates a network attack, said decision engine communicateswith at least one buffer to characterize the anomaly.
 2. The method ofclaim 1, wherein said normal base line values are generated by alearning mechanism comprising any of the following: day-timedifferential averaging or continuous IIR filtering/continuous averaging,and said method further comprising dynamically updating normal base linevalues over a predetermined time period and tuning a sensitivity of saidanomaly detection engine based on said updated normal base line values,and dynamically picking a learning mechanism from said day-hourdifferential averaging or said continuous IIR filtering/continuousaveraging based on a behavior of a network protected environment.
 3. Themethod of claim 1, wherein said method is implemented in a serveraccessible over a network and said network is any of, or a combinationof, the following: a local area network, a wide area network, theInternet, or a cellular network.
 4. The method of claim 1, wherein saidnetwork attack is a HTTP flood attack and said at least one of saidreal-time statistical parameters is HTTP request URL distribution. 5.The method of claim 4, wherein a remainder of said real-time statisticalparameters are any of, or a combination of, the following: number ofHTTP requests per second, HTTP outbound bandwidth per second, ratiobetween HTTP request to outbound HTTP bandwidth per second, number ofHTTP request per TCP connection, number of HTTP requests per second persource IP, and page errors/time-outs.
 6. The method of claim 2, whereinsaid at least one buffer further comprises: at least one source IP trapbuffer, said at least one source IP trap buffer detects abnormalrepetitions of HTTP request URL's per source IP address and protectedWeb server at least one HTTP request size trap buffer, said at least oneHTTP request trap buffer detecting abnormal repetitions of HTTP requestURL's per protected Web server only.
 7. The method of claim 1, whereinsaid at least one rate-based parameter is any of the following: a getrequest per second counter—Get_C, which is incremented by one uponclassification of any HTTP Get or Post request toward a protectedserver; an other request per second counter—Other_C, which isincremented by one upon classification of any HTTP request type that isnot Get or Post toward a protected server; a post request per secondcounter—Post_C, which is incremented by one upon classification of anyHTTP Post request toward a protected server; an outbound bandwidth persecond counter—OutB_C, which measures the outbound HTTP traffic that isgenerated by a protected server; a source HTTP request per secondcounter—S_R_C, which is attached to each source IP address that “owns”HTTP session toward a protected server and wherein the S_R_C counter isincremented by one upon classification of any HTTP Get or Post requesttoward a protected server; and a HTTP request per connectioncounter—Con_R_C, which is maintained per TCP connection and wherein theCon_R_C counter is incremented upon classification of any HTTP Get orPost request toward a protected server.
 8. The method of claim 7,wherein said at least one rate-invariant parameter is any of thefollowing: a ratio between outbound bandwidth to HTTP requests—Out/R_P,which measures an average outbound bandwidth per HTTP Get or Postrequests; and a HTTP request short-term size distribution table—S_D_T,which measures the current size distribution of HTTP request URL, andupon classification of a HTTP Get or Post requests, the HTTP Get or Postrequest being indexed.
 9. The method of claim 8, wherein said Out/R_Pratio parameter is calculated as follows:${{{Out}/{R\_ P}} = {{Min}\left( {\frac{OutB\_ C}{R\_ C},1000} \right)}},$where, OutB_C=outbound bandwidth and R_C=Get_C+Post_C.
 10. The method ofclaim 8, wherein HTTP request URL size index in said S_D_T is calculatedas follows: Request Size Index=INT(Request Size/10), where said RequestURL Size measures the length in bytes of a request URL path.
 11. Themethod of claim 10, wherein a measure of deviation from expectedRequest-URL Size is set over a predetermined time period.
 12. The methodof claim 1, wherein said learning mechanism maintains a long-term sizedistribution table—L_D_T, wherein a measure of deviation from expectedrequest size occurrences is set over a predetermined time period. 13.The method of claim 12, wherein a set of expected values E(i) incalculated from said L_D_T as follows: E(i)=P(i)×R_C_(n), where R_(—C)_(n) is the total number of HTTP request (Post and Get) in one secondand P(i) is the probability of each URL size grade.
 14. The method ofclaim 13, wherein, the normal, suspect and attack edges matrix arecalculated as follows:${{{Normal}\mspace{14mu} {edge}\text{:}\mspace{11mu} {N(i)}} = {E(i)}},{{{Attack}\mspace{14mu} {edge}\text{:}\mspace{11mu} {A(i)}} = \sqrt{l \times \left\lbrack {{E(i)} \times \left( {{E(i)} + {\sqrt{E(i)} \times P}} \right)} \right\rbrack}},{{{Suspect}\mspace{14mu} {edge}\text{:}\mspace{11mu} {S(i)}} = \sqrt{A \times N}},$where P is a confidence level, l is a configurable detection sensitivitylevel, and N is the number of entries in said L_D_T.
 15. The method ofclaim 1, wherein, after said characterization of an anomaly, any of thefollowing mitigation mechanisms are implemented: rate limit trafficflows which are defined by Source IP(s) and HTTP request URL(s) checksumtowards a protected server; rate limit traffic flows which are definedonly by HTTP request-URL checksum(s) toward a protected server; andblocking source IP(s) toward a protected server, wherein a rate limitfactor is set according to adapted HTTP request URL size distribution(E(i)).
 16. The method of claim 15, wherein said mitigation mechanismsis activated according to any of the following types of feedback: a DoAthat is generated according to all inbound traffic parameters prior tomitigation; and a DoA that is generated according to remaining trafficafter mitigation.
 17. A server-based architecture providing HTTP floodprotection comprising: a statistics module computing a plurality ofreal-time statistical parameters, said plurality of real-timestatistical parameters comprising at least one rate-based parameter andat least one rate-invariant and one rate-based parameter; a learningmodule computing normal base line values; an anomaly detection enginecomprising a storage area for storing embedded correlation rules and adegree of anomaly generator generating a degree of anomaly (DoA) basedon said received plurality of real-time statistical parameters and saidplurality of normal base line values; at least one source IP trapbuffer, said at least one source IP trap buffer storing a first list ofat least one source IP address that was identified based on an abnormalfrequency of HTTP request URL that was generated by said at least onesource IP address; and at least one HTTP request URL size trap buffer,said at least one HTTP request trap buffer storing a second list of atleast one suspicious HTTP request size deviating from an adapted sizedistribution base line, and wherein, when said generated degree ofanomaly indicates a network attack, said decision engine communicateswith said source IP trap buffer and said HTTP request URL size trapbuffer to characterize the anomaly.
 18. The server-based architecture ofclaim 17, wherein said real-time statistical parameters are any of, or acombination of, the following: number of HTTP requests per second, HTTPoutbound bandwidth per second, ratio between HTTP request to outboundHTTP bandwidth per second, HTTP request URL size distribution, number ofHTTP request per TCP connection, and number of HTTP requests per secondper source IP.
 19. The server-based architecture of claim 17, whereinsaid at least one rate-based parameter is any of the following: a getrequest per second counter—Get_C, which is incremented by one uponclassification of any HTTP Get or Post request toward a protectedserver; an other request per second counter—Other_C, which isincremented by one upon classification of any HTTP request type that isnot Get or Post toward a protected server; a post request per secondcounter—Post_C, which is incremented by one upon classification of anyHTTP Post request toward a protected server; an outbound bandwidth persecond counter—OutB_C, which measures the outbound HTTP traffic that isgenerated by a protected server; a source HTTP request per secondcounter—S_R_C, which is attached to each source IP address that “owns”HTTP session toward a protected server and wherein the S_R_C counter isincremented by one upon classification of any HTTP Get or Post requesttoward a protected server; and a HTTP request per connectioncounter—Con_R_C, which is maintained per TCP connection and wherein theCon_R_C counter is incremented upon classification of any HTTP Get orPost request toward a protected server.
 20. The server-basedarchitecture of claim 19, wherein said at least one rate-invariantparameter is any of the following: a ratio between outbound bandwidth toHTTP requests—Out/R_P, which measures an average outbound bandwidth perHTTP Get or Post requests; and a HTTP request short-term sizedistribution table—S_D_T, which measures the current size distributionof HTTP requests, and upon classification of a HTTP Get or Postrequests, the HTTP Get or Post request being indexed.
 21. Theserver-based architecture of claim 20, wherein said Out/R_P ratioparameter is calculated as follows:${{{Out}/{R\_ P}} = {{Min}\left( {\frac{OutB\_ C}{R\_ C},1000} \right)}},$where, OutB_C=outbound bandwidth and R_C=Get_C+Post_C.
 22. Theserver-based architecture of claim 20, wherein HTTP request URL sizeindex in said S_D_T is calculated as follows: Request SizeIndex=INT(Request Size/10), where said Request Size measures the lengthin bytes of a request URL path.
 23. The server-based architecture ofclaim 22, wherein a measure of deviation from expected Request URL Sizeis set over a predetermined time period.
 24. The server-basedarchitecture of claim 17, wherein said learning mechanism maintains along-term size distribution table—L_D_T, wherein a measure of deviationfrom expected request size occurrences is set over a predetermined timeperiod.
 25. The server-based architecture of claim 17, wherein, aftersaid characterization of an anomaly, any of the following mitigationmechanisms are implemented: rate limit traffic flows which are definedby Source IP(s) and HTTP request-URL(s) checksum towards a protectedserver; rate limit traffic flows which are defined only by HTTPrequest-URL checksum(s) toward a protected server; and blocking sourceIP(s) toward a protected server, wherein a rate limit factor is setaccording to adapted HTTP request URL size distribution (E(i)).